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Abstract 

Real world prediction problems often involve 
the simultaneous prediction of multiple tar- 
get variables using the same set of predictive 
variables. When the target variables are bi- 
nary, the prediction task is called multi-label 
classification while when the target variables 
are real-valued the task is called multi-target 
regression. Although multi-label classifica- 
tion can be seen as a specific case of multi- 
target regression, the recent advances in this 
field motivate a study of whether newer state- 
of-the-art algorithms developed for multi- 
label classification are applicable and equally 
successful in the domain of multi-target re- 
gression. In this paper we introduce two 
novel algorithms for multi-target regression, 
multi-target regressor stacking and regressor 
chains, inspired by two popular and success- 
ful multi-label classification approaches. Fur- 
thermore, we develop an extension of the re- 
gressor chains algorithm which aims at im- 
proving its predictive performance and which 
is also applicable in the classification domain. 
All methods are empirically evaluated on 6 
multi-target regression data sets, 4 of which 
are firstly introduced in this paper. The re- 
sults of the evaluation show that all the pro- 
posed multi-target methods are able to im- 
prove the accuracy of the baseline approach 
which performs a separate regression for each 
target and that the corrected regressor chains 
method achieves the best overall accuracy. 



1. Introduction 

Learning from multi-label data has recently received 
increased attention by researchers working on machine 
learning and data mining for two main reasons. The 
first one is the ubiquitous presence of multi-label data 
in application domains ranging from multimedia infor- 
mation retrieval to tag recommendation, query catego- 
rization, gene function prediction, medical diagnosis, 
drug discovery and marketing. The other reason is a 
number of challenging research problems involved in 
multi-label learning, such as dealing with label rarity, 
scaling to large number of labels and exploiting label 
relationships (e.g. hierarchies), with the most promi- 
nent one being the explicit modeling of label depen- 
dencies (Dembczyhski et al., 2012). 

Multi-label learning is closely related to multi-target 
regression, also known as multivariate or multi-output 
regression, which aims at predicting multiple real- 
valued target variables instead of binary ones. Despite 
that multi-target regression is a less popular task, it 
still arises in several interesting domains, such as pre- 
dicting the wind noise of vehicle components (Kuznar 
et al., 2009), stock price prediction and ecological mod- 
eling (Kocev et al., 2009). Multi-label learning is often 
treated as a special case of multi-target regression in 
statistics (Izenman, 2008). However, we could more 
precisely state that both are instances of the more 
general learning task of predicting multiple targets, 
which could be real-valued, binary, ordinal, categori- 
cal or even of mixed type. The baseline approach of 
learning a separate model for each target applies to 
both learning tasks. Furthermore, there exist tech- 
niques, such as (Blockeel et al., 1998; Seeger et al., 
2005; Lebanon & Balasubramanian, 2012), that can 
naturally handle both tasks. Most importantly, they 
share the same core challenge of modeling dependen- 
cies among the different targets. 
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Given this tight connection between these two learning 
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tasks, it would be interesting to investigate whether 
recent advances in the more popular multi-label learn- 
ing task can be successfully transferred to multi-target 
regression. In particular, this paper adapts two multi- 
label learning methods that successfully model label 
dependencies (Godbolc & Sarawagi, 2004; Read et al., 
2011) to multi-target regression. The results of an 
empirical evaluation on several real-world data sets, 
some firstly introduced here, show that the benefits 
of these two multi-label algorithms apply also to the 
multi-target regression setting. 

Furthermore, by studying the relationship between 
these two learning tasks, this work aims to increase 
our understanding in their shared challenge of mod- 
eling target dependencies and widen the applicability 
of existing specialized techniques for either task. This 
kind of abstraction of key ideas from solutions tailored 
to related problems offers additional advantages, such 
as improving the modularity and conceptual simplicity 
of learning techniques and avoiding reinvention of the 
same solutions 1 . 

The rest of the paper is organized as follows: Sec- 
tion 2 discusses related work in the field of multi-label 
classification and gives some insight into which multi- 
label classification methods would be more appropriate 
for multi-target regression. Section 3 gives detailed 
descriptions of the proposed methods and Section 4 
presents the evaluation methodology and introduces 
the data sets used in the empirical evaluation. The ex- 
perimental results are shown in Section 5 and finally, 
Section 6 concludes our study. 

2. Prom Multi-Label Classification to 
Multi- Target Regression 

Multi-label learning methods are often categorized into 
those that adapt a specific learning approach (e.g. k 
nearest neighbors, decision tree, support vector ma- 
chine) for handling multi-label data and those that 
transform the multi-label task into one or more single- 
label tasks that can be solved with off-the-shelf learn- 
ing algorithms (Tsoumakas et al., 2010). The latter 
can be further categorized to those that model single 
labels, pairs of labels and sets of labels (Zhang, 2011). 

Approaches that model single labels include the typical 
one-versus-all (also known as binary relevance) base- 
line, methods based on stacked generalization (God- 
bole & Sarawagi, 2004; Tsoumakas et al., 2009; Cheng 
& Hiillermeier, 2009) and the classifier chains algo- 

1 See the motivation of the NIPS 2011 workshop on re- 
lations among machine learning problems at http : / /rml . 
ami . edu . au/ 



rithm (Read et al., 2011; Dcmbczyhski et al., 2010). 
Such approaches are almost straightforward to adapt 
to multi-target regression by employing a regression 
instead of a classification algorithm. It is this kind of 
approaches that we extend in this work. 

Approaches that model pairs of labels (Fiimkranz 
et al., 2008) follow the paradigm of the one-versus- 
one decomposition for using binary classifiers on multi- 
class learning tasks. This concept however is not trans- 
ferable to multi-target regression. The same holds for 
approaches that consider sets of labels as different val- 
ues of a single-label multi-class task (Read et al., 2008; 
Tsoumakas et al., 2011a). 

The extension of multi-label algorithm adaptation 
methods for handling multi-target data mainly de- 
pends on how easy it is for the underlying learning 
algorithm to handle classification and regression data 
interchangeably. For example, decision trees can han- 
dle both classification and regression data through dif- 
ferent functions for calculating the impurity of internal 
nodes and the output of leaves. It thus comes as no 
surprise that there exist decision tree algorithms for 
both multi-label classification (Clare & King, 2001) 
and multi-target regression (Appice & Dzeroski, 2007), 
with the most representative and well-developed ones 
being based on the predictive clustering trees frame- 
work (Blockeel et al., 1998; Kocev et al., 2007). It 
is interesting to note that the predictive clustering 
framework is an example of a technique that originally 
focused on multi-target regression, and only recently 
showcased its effectiveness on multi-label classification 
(Vens et al., 2008; Madjarov et al., 2012). 

Several multi-label algorithm adaptation methods are 
based on the definition and optimization of alterna- 
tive loss functions, compared to the algorithms they 
extend. For example, the core idea in (Zhang & Zhou, 
2006) was the optimization of a ranking loss function 
that takes into account label pairs, instead of the typ- 
ical logistic loss of neural networks that looks at in- 
dividual labels. Similarly, (Crammer & Singer, 2003) 
defines a family of online algorithms based on different 
loss functions considering label relationships. To the 
best of our knowledge, this type of reasoning for algo- 
rithm design has not been transferred to multi-target 
regression. We believe that it could be an interesting 
avenue to investigate. For example, consider an appli- 
cation of food sales prediction (Zliobaite et al., 2009), 
in particular pastry sales prediction for a patisserie in 
order to minimize the amount of pastries with short 
expiration date that are thrown away. In this appli- 
cation, we may like to minimize the sum of prediction 
errors, but we might also want to minimize the maxi- 
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mum individual prediction error, to avoid for example 
an early run-out of any of the pastries. 

Finally, we would like to note that there exist ap- 
proaches, such as (Seeger et al., 2005; Lebanon & Bala- 
subramanian, 2012), that have recognized this dual ap- 
plicability (classification and regression) of their multi- 
target approach and have given a general formulation 
of their key ideas. 

3. Methods 

We first formally describe the multi-target regression 
task and provide the notation that will be used next 
for the description of the methods. Let X and Y be 
two random vectors where X consists of d predictive 
variables Xi,..,Xd and Y consists of m target vari- 
ables Yx , . . , Y m . We assume that samples of the form 
(x, y) are generated iid -by some source- according to a 
joint probability distribution P(X, Y) on X x y where 
X = R d and y = R m are the domains of X and Y 
and are often referred to as the input and the out- 
put space. In a sample (x,y), x = [xi,..,Xd] is the 
input vector and y = [yi,..,y rn ] is the output vector 
which are realizations of X and Y respectively. Given 
a set D = {(x 1 , y 1 ), .., (x n , y n )} of n training exam- 
ples, the goal in multi-target regression is to learn a 
model h : X — > y which given an input vector x q , is 
able to predict an output vector y q = h(x q ) which best 
approximates the true output vector y q . 

In the baseline Single Target (ST) method, a multi- 
target model h is comprised of m single-target models 
hj : X — > R where each model hj is trained on a 
transformed training set Dj — {(x 1 , yj), .., (x™, y™)} to 
predict the value of a single target variable Yj. This 
way, target variables are predicted independently and 
potential relations between them cannot be exploited. 

3.1. Multi- Target regressor stacking 

The first method that we consider is inspired from 
(Godbolc & Sarawagi, 2004) where the idea of stacked 
generalization was applied in a multi-label classifica- 
tion context. Here, we adapt this problem transfor- 
mation method for multi-target regression and de- 
note it as MTRS (Multi-Target Regressor Stacking). 
The training of MTRS consists of two stages. In the 
first stage, m single-target models hj : X — > R are 
learned as in ST. However, instead of directly us- 
ing these models for prediction, MTRS involves an 
additional training stage where a second set of m 
meta models h* : X x y — > R are learned, one for 
each target Yj. Each meta model h* is learned on a 
transformed training set D* = {(x* 1 ,?/]), .., (x*",y™)} 



where x* 1 = [x\, .., x % n ,y\, .., are transformed input 
vectors consisting of the original input vectors of the 
training examples augmented by predictions (or esti- 
mates) of their target variables obtained by the first 
stage models. MTRS is based on the idea that a sec- 
ond stage model is able to correct the prediction of a 
first stage model by using information about the pre- 
dictions of other first stage models. 

To obtain predictions for an unknown instance x q , 
the first stage models are first applied and an out- 
put vector y q = [y\,..,y q m ] = [h^x"), .., h m {x q )} 
is obtained. Then the second stage models are 
applied on a transformed input vector x* q = 
[x\, .., x^, yf, to produce the final output vector 

f = [ht(x* q ),...,h* m (x* g )}- 

An important issue in MTRS is how the meta training 
sets D* should be created and specifically how the pre- 
dictions of the values of the target variables of the meta 
training examples should be obtained. In (Godbole & 
Sarawagi, 2004), the first stage models are learned us- 
ing the full set of training examples and these models 
are applied again on all the training examples to ob- 
tain the predictions used by the second stage models. 
This approach, however, violates a core assumption 
in supervised learning, namely that the training and 
testing data should be identically and independently 
distributed. The in-sample estimates which are used 
to form the second stage training examples are ex- 
pected to follow a different distribution than the out- 
of-sample estimates which will be obtained at predic- 
tion time. 

An alternative approach is to use only a part of the 
training examples for learning the first stage models 
and applying the learned models on the holdout part in 
order to form the meta training sets. While the distri- 
bution of the estimates obtained using this approach is 
expected to be more representative of the distribution 
of the estimates obtained during prediction, it leads 
to a reduced meta training set as only the examples 
of the holdout set can be used for training the second 
stage models. Here, we follow an approach that uses 
all the available training examples in both stages and 
at the same time avoids the aforementioned problem. 
This is achieved by applying an internal f-fold cross- 
validation to obtain predictions for the target variables 
of all the training examples. These cross-validation es- 
timates, are expected to most accurately approximate 
the distribution of the estimates obtained in prediction 
phase. 
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3.2. Regressor Chains 

The second method that we consider is inspired by the 
recently proposed (Read ct al., 2011) Classifier Chains 
(CC) method and we henceforth denote it as Regres- 
sor Chains (RC). RC is another problem transforma- 
tion method, based on the idea of chaining single- 
target models. The training of RC consists of se- 
lecting a random chain (permutation) of the set of 
target variables and then building a separate regres- 
sion model for each target. Assuming that the de- 
fault chain C — {Y x , Y%, .., Y m } (where C is repre- 
sented as an ordered set) is selected, the first model 
which concerns the prediction of Y\, has the form 
h\ : X — > R and it is same as the model built by 
the ST method for this target. The difference in RC 
is that subsequent models hjjyi are trained on trans- 
formed datasets D* = {{x* 1 , yj), .., (x* n , ?/")}, where 
x* 1 = [x\, .. ,x\, y\, .., Vj_i] are transformed input vec- 
tors consisting of the original input vectors of the train- 
ing examples augmented by the actual values of all pre- 
vious targets in the chain. Thus, the models built for 
targets j>i have the form hj : XxR 3 —> R. Given 
such a chain of models, the output vector y q of an un- 
known instance x q is obtained by sequentially applying 
the models hj, thus y q = [hi(x q ), h2{x^), .., h m (x^ q )] 
where x* q j >1 = [x\ , .., x q d , y\ , .., Note that since 

the true values y\, ..,yi_i of the target variables are 
not available at prediction time, the method relies 
on estimates of these values obtained by applying the 
models hi,.., ftj-i. 

3.3. Regressor Chains Corrected 

Despite its notable success in practical multi-label clas- 
sification problems and its theoretical grounding as 
shown in (Dcmbczyhski et al., 2010), we notice a lim- 
itation of the original CC (and consequently the RC) 
method which has not yet been pointed out in the 
multi-label literature and propose a new training pro- 
cess that aims to fix it. In the training phase, RC 
uses the actual target values of the training examples 
for learning the chained models while it relies on esti- 
mates of these values at prediction time. This particu- 
lar choice violates the iid assumption, as the distribu- 
tion of the actual target values will be different from 
the distribution of the estimated values. We observe 
that the problem is similar to that of the multi-label 
stacking method of (Godbole & Sarawagi, 2004). In 
fact, the situation is expected to be even more prob- 
lematic in the case of RC as the distribution of the ac- 
tual target values can diverge even more radically from 
the distribution of the predicted values (compared to 
the in-sample estimates used in (Godbole & Sarawagi, 
2004)). The impact that this divergence in the distri- 



butions has on model accuracy can depend on several 
factors such as: a) magnitude of the divergence, b) 
chain order and length, c) degree and nature of la- 
bel dependency. A systematic study of these factors 
is out of the scope of this paper where we focus on 
presenting an extension to the original method which 
corrects this problem. In Regression Chains Corrected 
(RCC) we use the predictions of the regression models 
earlier in the chain instead of the true values of the 
target variables. These predictions are obtained using 
an internal f-fold cross-validation approach similar to 
the one used in MTRS. This corresponds to the first 
model trained as in RC but subsequent models trained 
on datasets D* = {(x* 1 , yj), .., (x* n , y™)}, where this 
time x* 1 — [x\, ..,x\,y\, ..,?/*•_■[], i.e. the original input 
vectors are augmented by cross-validation estimates 
instead of the actual values. Predictions for unknown 
instances are obtained as in the original RC method. 

One notable property of RC / RCC methods is that 
they are sensitive in the selected chain ordering. The 
main problem arising from the use of a single ran- 
dom chain, is that targets which appear earlier in a 
chain cannot model potential statistical relationships 
with targets appearing later in that chain. Addition- 
ally, prediction error is likely to propagate and increase 
along a chain when making predictions for a new test 
instance (this affects mainly the RC method) . To mit- 
igate these effects, (Read et al., 2011) proposed an 
ensemble scheme called Ensemble of Classifier Chains 
where a set of CC models with differently ordered 
chains are built on bootstrap samples of the training 
set and the final predictions come from majority vot- 
ing. This scheme has proven to consistently improve 
the accuracy of a single CC in the classification do- 
main. We apply the same idea (without sampling) 
on our regression variants and call the resulting ap- 
proaches Ensemble of Regressor Chains (ERC) and 
Ensemble of Regressor Chains Corrected (ERCC). 

4. Experiments 

In this empirical evaluation we compare the accuracy 
of the proposed methods (MTRS, ERC, ERCC) with 
the accuracy of the ST baseline on 6 benchmark multi- 
target regression data sets which are presented in the 
following subsection. Accuracy is measured in terms of 
Relative Root Mean Squared Error (RRMSE), where 
RRMSE for a target variable is equal to the RMSE 
divided by the RMSE of predicting the average value 
of that target in the training set. To test the statis- 
tical significance of the observed differences between 
the methods, we follow the methodology suggested in 
(Demsar, 2006) for the comparison of multiple meth- 
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ods on multiple datasets. We first apply the non- 
parametric Friedman test which operates on the av- 
erage ranks of the methods and checks the validity of 
the hypothesis (null-hypothesis) that all methods are 
equivalent. If the null-hypothesis is rejected, we pro- 
ceed to the Nemenyi post-hoc test that makes pair- 
wise comparisons and computes the critical distance 
(between the average ranks) required in order for two 
methods to be considered significantly different. Fi- 
nally, we graphically present the results with appro- 
priate diagrams which plot the average ranks of the 
methods and show groups of methods whose average 
rank differences are less than the critical distance for 
a p- value of 0.1. As the above methodology requires a 
single performance measurement for each method on 
each data set, it is not directly applicable to multi- 
target evaluation where we have multiple performance 
measurements, one for each target, for each method 
on each data set. One option is to take the average 
RRMSE across all target variables within a data set 
as a single performance measurement. However, this 
may not always be a meaningful choice since: a) dif- 
ferent targets may represent different things and b) 
we may not be always interested into the best aver- 
age performance (see patisserie example in Section 2). 
Another option is to treat the RRMSE performance 
of each method on each different target as a differ- 
ent measurement. In this case, however, Friedman's 
test assumption of independence between performance 
measurements is violated. In the absence of a better 
solution, we perform the two dimensional analysis of 
(Aho et al., 2009), where statistical tests are conducted 
taking both the average RRMSE values per data set, 
but also considering the RRMSE value per target as an 
independent measurement. RRMSE is estimated us- 
ing either 10-fold cross-validation or a single train-test 
split evaluation. As explained in the next subsection, 
the time-dependent relationship among the examples 
of some datasets, makes the use of cross-validation in- 
appropriate as it would entail information leak. 

Since all the evaluated methods transform the multi- 
target prediction problem into several single-target 
problems, they can be coupled with any off-the-shelf 
regression algorithm. In preliminary experiments us- 
ing 4 well-established regression algorithms: 1) Ridge 
regression (Hoerl & Kennard, 1970), 2) SVM regres- 
sion (Drucker et al., 1997), 3) Bagging (Brciman, 1996) 
of regression trees and 4) Stochastic Gradient Boosting 
(Friedman, 1999) we found that no single algorithm 
performs better than the other algorithms under all 
settings. Thus, instead of arbitrarily choosing a single 
regression algorithm, we treat the base regression al- 
gorithm as a parameter of the evaluated multi-target 



methods and report the best performance obtained by 
each method using any of the 4 algorithms. We believe 
that this choice represents a more realistic scenario and 
allows us to conclude about the relative performance 
of the multi-target methods independently of the se- 
lection of the base regression algorithm. 

All multi-target methods and the evaluation frame- 
work are implemented as an extension to Mulan 2 
(Tsoumakas et al., 2011b) library for multi-label learn- 
ing. This allowed us to use existing Weka's imple- 
mentations of the base regression algorithms. Specif- 
ically, for Ridge regression we used the LinearRegres- 
sion class with options for attribute selection and 
elimination of collinear attributes turned off. To 
find the best value for the regularization parameter 
{10~ 4 , 10 -3 , . . . , 10 3 , 10 4 } in each single-target predic- 
tion problem, we performed internal 10-fold cross- 
validation and selected the single-target model with 
the lowest root mean squared error (RMSE) . For SVM 
regression we used Weka's wrapper class for LIBSVM 
(Chang & Lin, 2011) and selected nu-SVR as the SVM 
type with default parameters. For Bagging of regres- 
sion trees we coupled the Bagging class with the REP- 
Tree class (Weka's implementation of regression tree) 
and used 10 bootstrap samples. Finally for Stochas- 
tic Gradient Boosting we used Weka's AdditivcRcgres- 
sion class coupled with a Decision Stump and fixed the 
number of boosting iterations at 100. 

We use / = 10 internal cross-validation folds in both 
MTRS and ERCC and an ensemble size of 10 models 
in both ERC and ERCC, each one trained using a 
different random chain (in cases where the number of 
distinct chains is smaller than 10 we create exactly as 
many models as the number of distinct label chains) . 
In order to ensure a fair comparison of ERC and ERCC 
with the non-ensemble methods, we do not perform 
bootstrap sampling (i.e. each ensemble model is build 
using all training examples). 

4.1. Description of Data Sets 

Despite the numerous interesting applications of multi- 
target regression, there are few publicly available data 
sets of this kind, perhaps because most applications 
are industrial. In fact, among the data sets used in 
other multi-target regression studies, e.g. (Aho et al., 
2009), (Zenko & Dzeroski, 2008), only Solar Flare 3 
and Water Quality 4 (Dzeroski et al., 2000) are publicly 

2 http : / /mulan. sourcef orge .net 

3 http : / /archive . ics .uci . edu/ml/datasets/Solar+ 
Flare 

http : / /kt . i js . si/DragiKocev/PhD/resources/ 
doku. php?id=multi_target_regression 
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available. This lack of available data, motivated the 
collection of real-world multi-target regression data 
and the composition of new benchmark data sets 5 . A 
description of each data set follows, while Table 1 sum- 
marizes their statistics. 

Table 1. Statistics of the data sets used in the evaluation. 
Data sets having two values for Examples indicate the 
sample counts for training and testing, respectively, d de- 
notes the number of descriptive attributes, m denotes the 
number of targets. Datasets in bold are firstly introduced 
in this paper. 



Data set 


Examples 


d 


m 


Solar Flare 


323/CV 


26 


3 


Water Quality 


1060/CV 


16 


14 


OES97 


334/CV 


263 


16 


OES10 


403/CV 


298 


16 


ATP 


188/108 


411 


6 


RF 


4156/5065 


64 


8 



Solar Flare: This data set concerns the prediction of 
the number of times a certain type (C common, M 
moderate, X severe) of solar flare (sudden brighten- 
ing observed over the sun surface or the solar limb) 
occurred in a 24 hour period. Note that multi-valued 
nominal attributes in the original data set were re- 
placed by a set of binary attributes, one for each dis- 
tinct value of the original attributes. After this trans- 
formation the data set has 26 descriptive attributes. 

Water Quality: This data set concerns the prediction 
of the abundance of 14 plant and animal species in 
Slovenian rivers, using measurements of 16 physical 
and chemical water quality parameters. 

OES: These data sets were obtained from years 1997 
(OES97) and 2010 (OES10) of the annual Occupa- 
tional Employment Survey compiled by the US Bu- 
reau of Labor Statistics 6 . Each row provides the esti- 
mated number of full-time equivalent employees across 
many employment types for a specific metropolitan 
area. There are 334 and 403 cities in the 1997 and 
May 2010 data sets, respectively. The input variables 
in these datasets are a randomly sequenced subset of 
employment types (i.e. doctor, dentist, car repair tech- 
nician) observed in at least 50% of the cities (some 
categories had no values for particular cities). The 
targets for both years are randomly selected from the 

5 A11 data sets can be downloaded from http: //users. 
auth.gr/espyromi/datasets .html. 

6 The raw data can be obtained at the Bureau of Labor 
Statistics web site: http://www.bls.gov/. 



entire set of categories above the 50% threshold. Miss- 
ing values in both the input and the target variables 
were replace by sample means for these results. To our 
knowledge, this is the first use of the OES data set for 
benchmarking of multi-target prediction algorithms. 

ATP: The airline ticket price data set is temporal in 
nature. The rows are a sequence of time-ordered ob- 
servations over several days. Each sample in this data 
set represents a set of observations from a specific ob- 
servation date and departure date pair. The input 
variables for each sample are values that may be use- 
ful for prediction of the airline ticket prices for a spe- 
cific departure date. The target variables in this data 
set is the minimum price observed over the next 7 
days for 6 target flight preferences (any airline with 
any number of stops, any airline non-stop only, Delta 
Airlines, Continental Airlines, Airtrain Airlines, and 
United Airlines). The input variables include the fol- 
lowing types: the number of days between the obser- 
vation date and the departure date (1 feature), the 
boolean variables for day-of-the-week of the observa- 
tion date (7 features), the complete enumeration of the 
following 4 values: 

• the minimum price, mean price, and number of 
quotes from 

• all airlines and from each airline quoting more 
than 50% of the observation days 

• for non-stop, one-stop, and two-stop flights 

• for the current day, previous day, and two days 
previous. 

The result is a feature set of 411 variables. For specific 
details on how this data set is constructed please con- 
sult (Groves & Gini, 2011). The nature of this data 
set is heterogenous with a mixture of several types 
of variables including boolean variables, prices, and 
counts. Because of the temporally dependent relation- 
ship of the examples in this data set, it is not per- 
missible to reorder the observations when performing 
conventional cross-validation. Additionaly, perform- 
ing conventional cross-validation risks leaking informa- 
tion between the training set and test set which can 
lead to artificially good performance for some learning 
algorithms. For this reason, this data set is split tem- 
porally at a specific observation date for training and 
testing: examples with an observation date before the 
split date are in the training set, all other examples 
are in the test set. 

RF: The river flow domain is a temporal prediction 
task designed to test predictions on the flows in a river 
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network for 48 hours in the future at specific locations. 
The data set contains data from hourly flow observa- 
tions for 8 sites in the Mississippi River network in 
the United States and were obtained from the US Na- 
tional Weather Service. Each row includes the most 
recent observation for each of the 8 sites as well as 
lagged observations from 6, 12, 18, 24, 36, 48 and 60 
hours in the past. Each gauge site contributes 8 at- 
tribute variables to facilitate prediction. There are 
a total of 64 variables plus 8 target variables in the 
data set. The data set contains over 1 year of hourly 
observations (>9,000 hours) collected from September 
2011 to September 2012. The training set is the first 
40% of observations, and the test set is the remainder. 
The domain is a natural candidate for multi-target re- 
gression because there are clear physical relationships 
between readings in a contiguous river network. 



performance of ERCC is significantly better than that 
of ST. On the other hand, the experimental data is not 
sufficient to reach statistically significant conclusions 
regarding the performance of MTRS and ERC. 
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5. Results 

Table 2 shows the average RRMSE obtained by each 
multi-target method on each data set. We notice that 
the ERCC method obtains the lowest rank in both the 
per data set and the per target analysis, followed by 
MTRS in the per data set and by ERC in the per target 
analysis. In both cases, all the proposed methods are 
ranked above the baseline. As the per target results 
reveal, the baseline approach still obtains the lowest 
error in 9 out of 63 targets. 

Table 2. The table shows the average RRMSE obtained by 
each multi-target method on each data set. In each row, 
the lowest error is typeset in bold. The two last rows show 
the average rank of each method calculated over data set 
RRMSE averages (D) and over all targets (T). 



Figure 1. Comparison of all methods with the Nemenyi 
test calculated on average RRMSE per dataset. Groups of 
methods that are not significantly different (at p = 0.10) 
are connected. 
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The Friedman test shows significant differences be- 
tween the methods at p = 0.01 when the analysis is 
performed per data set and at p = 0.05 when the anal- 
ysis is performed per target. Thus, in both cases we 
proceed to the Nemenyi post-hoc test, whose results 
at p = 0.1 are presented in the average rank diagrams 
of Figures 1 and 2. In both diagrams, we see that the 



Figure 2. Comparison of all methods with the Nemenyi test 
calculated on RRMSE per target. Groups of methods that 
are not significantly different (at p = 0.10) are connected. 



6. Conclusions and Future Work 

This paper attempts to exploit the connection between 
multi-label classification and multi-target regression 
by investigating the applicability of methods proposed 
to solve the former task into the later task. The analy- 
sis of Section 2 reveals that transformation multi-label 
methods which model each label independently are 
directly applicable to multi-target regression, simply 
by employing regression models. In Section 3 we pre- 
sented two such methods, MTRS and RC along with 
a corrected version of the RC method (RCC) which 
are applied for the first time in the domain of multi- 
target regression. As revealed by the empirical eval- 
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uation, by explicitly modeling label dependencies, the 
proposed methods are able to improve the performance 
of the baseline approach in most targets with ERCC 
performing significantly better than the baseline ap- 
proach. Another important contribution of this paper 
is the introduction of several new benchmark multi- 
target datasets. 

As future work, we would like to perform a deeper 
analysis into why different multi-target methods per- 
form better on different datasets with particular inter- 
est on the kind of dependencies between target vari- 
ables that are most prominent in each data set. Fi- 
nally, we would like to compare the proposed meth- 
ods with other state-of-the-art multi-target regression 
approaches, such as those falling into the predictive 
clustering framework. 
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